Parallel storage systems are widely used in many computing environments. Parallel storage systems provide high degrees of concurrency in which many distributed processes within a parallel application simultaneously access a shared file namespace.
Parallel computing techniques are used in many industries and applications for implementing computationally intensive models or simulations. For example, the Department of Energy uses a large number of distributed compute nodes tightly coupled into a supercomputer to model physics experiments. In the oil and gas industry, parallel computing techniques are often used for computing geological models that help predict the location of natural resources. One particular parallel computing application models the flow of electrons within a cube of virtual space by dividing the cube into smaller sub-cubes and then assigning each sub-cube to a corresponding process executing on a compute node.
Storage tiering techniques are increasingly used in parallel computing environments to more efficiently store the vast amounts of information. For example, the Symmetrix system from EMC Corporation is an enterprise storage array that optionally includes Fully Automated Storage Tiering (FAST). Storage tiering techniques typically combine Non-Volatile Random Access Memory (NVRAM), also referred to as flash memory, with more traditional hard disk drives (HDDs). Flash memory is used to satisfy the bandwidth requirements of a given system while the hard disk drives are used to satisfy the capacity requirements. Existing storage systems often store the data by serializing the data into linear arrays of bytes that are stored as sub-files. Metadata is stored with each file including a memory offset and length that indicate where the sub-file fits into the larger logical file.
A need exists for improved techniques for storing sub-files with semantically meaningful boundaries. A further need exists for improved techniques for storing sub-files with metadata that includes semantic information.